Forward Progress Mechanisms for a Communications Network having Multiple Nodes

ABSTRACT

A communication network includes: a plurality of nodes in a topology, with each node having an upstream and a downstream neighboring node in the topology; a separate unidirectional communication link coupled between each node and that node&#39;s downstream neighboring node; and a separate unidirectional control link coupled between each node and that node&#39;s upstream neighboring node. A controller in each node keeps a count of packets sent by that node via the corresponding unidirectional communication link. The controller uses the count of packets sent to determine whether a given packet is allowed to be sent from that node to the downstream neighboring node and, if so, whether a full rate or a throttled rate is to be used for sending the given packet. Based at least in part on the determining, the controller selectively sends the given packet to the downstream neighboring node.

BACKGROUND Related Art

Some electronic communications networks include multiple nodes (e.g.,computing devices, network devices, etc.) that are arranged in atopology such as a ring, a mesh, or a star. In some of thesecommunication networks, each node has an upstream neighboring node and adownstream neighboring node in the topology. Each node is connected toits upstream and downstream neighboring nodes via separateunidirectional communication links (e.g., serial or parallel buses orsignal lines). The nodes communicate with other nodes by sending andforwarding packets (or messages, data units, bit sequences, etc.) on theunidirectional communication links between the nodes. More specifically,when sending a packet to a destination node in a communication network,a source node sends the packet to the source node's downstreamneighboring node via the corresponding unidirectional communicationlink. When the downstream neighboring node is the destination node, thedownstream neighboring node processes the packet. Otherwise, thedownstream neighboring node forwards the packet to its own downstreamneighboring node via the corresponding unidirectional communicationlink. In this way, the packet is forwarded from node to node though thetopology until reaching the destination node.

In some of the above-described communication networks, theunidirectional communication links are only able to handle one packetper cycle of a controlling clock (i.e., a clock in the communicationnetwork via which the nodes are synchronized). When a node is forwardinga packet received from an upstream neighboring node in a given clockcycle, therefore, the node is unable to send the node's own packetduring the given clock cycle. This can lead to partial or complete“starvation” of nodes, and thus an unfair/undesirable balance of theconsumption of bandwidth in the communication network, when one or moreupstream nodes are sending packets in many or all of the cycles of thecontrolling clock. In other words, a node can be blocked from sendingits own packets when upstream nodes are sending sufficient packets tooccupy the available clock cycles.

In order to reduce the effects of starvation and in an attempt to ensurethat nodes in communication networks are able to use specifiedpercentages of clock cycles for sending packets (e.g., equal percentagesfor all nodes), designers have proposed using a “control token” toenable the nodes to send packets in situations in which upstream nodesmight otherwise block the nodes from sending packets. Generally, in suchcommunication networks, separate unidirectional control links areconnected between each node and its upstream neighboring node in thetopology. The control token, which can be as simple as a single binarybit, is passed from node to node to make trips through the topology in adirection in reverse of the direction in which packets are sent. Eachnode maintains a count of packets sent and operates under a rule that atmost a maximum threshold value worth of packets (e.g., 16, 20, etc.) canbe sent between the last time that the control token was passed on fromthat node and the control token arriving back at that node. In otherwords, while the control token makes a trip through all the nodes in thetopology, that node can send packets until the count of packets sentequals the maximum threshold value. Each node also operates under a rulethat, when that node receives the control token from a downstreamneighboring node, when that node has no packets to send or that node hasalready sent a minimum threshold value worth of packets (e.g., 8, 12,etc.), that node passes the control token to that node's upstreamneighboring node in a next clock cycle. Thus, the control token may bepassed from node to node as rapidly as in each clock cycle. When passingthe control token to an upstream neighboring node, each node resets itscount of packets sent, which frees that node to send another maximumthreshold value worth of packets while the control token makes thesubsequent trip through the nodes in the topology. In contrast, whenthat node receives the control token from a downstream neighboring node,has packets to send, and has not already sent the minimum thresholdvalue of packets, that node holds the control token (i.e., does not passthe control token to the upstream neighboring node in the next clockcycle). While holding the control token, that node is free to sendpackets until that node's count of packets sent is equal to the minimumthreshold value. By holding the control token, that node prevents thecontrol token from passing through the nodes in the topology andenabling the reset of counts of packets sent in the other nodes. Theother nodes will therefore send up to the maximum threshold value ofpackets and then stop sending packets. Even where one or more upstreamnodes are sending packets, each upstream neighboring node eventuallysends at most the maximum number of packets before stopping sendingpackets. This means that the one or more nodes that are preventing thatnode from sending packets are stopped, at some point, from sendingpackets, which frees the node that holds the control token to send up tothe minimum number of packets before passing the control token on tothat node's upstream neighboring node. One example of such acommunication network is described by I. Cidon and Y. Ofek in MetaRing—AFull-Duplex Ring with Fairness and Spatial Reuse, IEEE Transactions onCommunications, Vol. 41, January 1993.

Although using the control token can help to ensure that all nodes makesome forward progress—i.e., are each given a chance to send at leastsome packets—nodes can still experience inefficient operation when usingthe control token. For example, because holding the control token caneventually stop all nodes in a topology from sending packets, nodes thatare not near a starved node (e.g., on the “other side” of a largertopology), are not involved in the starvation of the starved node,and/or are not in a communication range of the starved node (e.g., arefurther than a specified number of nodes away from the starved node inthe topology) may be stopped from sending packets due to the starvednode holding the control token. The existing use of control tokens istherefore suboptimal.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 presents a block diagram illustrating a node in accordance withsome embodiments.

FIG. 2 presents a block diagram illustrating nodes in a topology inaccordance with some embodiments.

FIG. 3 presents a flowchart illustrating a process for sending packetsfrom a node in a topology in accordance with some embodiments.

FIG. 4 presents a flowchart illustrating a process for handling acontrol token in a node in a topology in accordance with someembodiments.

FIG. 5 presents a flowchart illustrating forwarding packets from a nodein a topology in accordance with some embodiments.

FIG. 6 presents a swim lane diagram illustrating communications of nodesin a topology in accordance with some embodiments.

FIG. 7 presents a block diagram illustrating nodes in a first topologyand nodes in a second topology with a control token crossover link inaccordance with some embodiments.

Throughout the figures and the description, like reference numeralsrefer to the same figure elements.

DETAILED DESCRIPTION

The following description is presented to enable any person skilled inthe art to make and use the described embodiments and is provided in thecontext of a particular application and its requirements. Variousmodifications to the described embodiments will be readily apparent tothose skilled in the art, and the general principles defined herein maybe applied to other embodiments and applications. Thus, the describedembodiments are not limited to the embodiments shown, but are to beaccorded the widest scope consistent with the principles and featuresdisclosed herein.

Terminology

In the following description, various terms are used for describingembodiments. The following is a simplified and general description ofone of these terms. Note that this term may have significant additionalaspects that are not recited herein for clarity and brevity and thus thedescription is not intended to limit this term.

Functional block: functional block refers to a group, collection, and/orset of one or more interrelated circuit elements (or “circuitry”) suchas integrated circuit elements, discrete circuit elements, etc. Thecircuit elements are “interrelated” in that circuit elements share atleast one property. For example, the interrelated circuit elements maybe included in, fabricated on, or otherwise coupled to a particularintegrated circuit chip or portion thereof, may be involved in theperformance of given functions (computational or processing functions,memory functions, etc.), may be controlled by a common control elementand/or a common clock, etc. A functional block can include any number ofcircuit elements, from a single circuit element (e.g., a singleintegrated circuit logic gate) to millions or billions of circuitelements (e.g., an integrated circuit memory).

Overview

In the described embodiments, an electronic communication networkincludes multiple nodes (e.g., computing devices, network devices,sensors, etc.) arranged in a topology (e.g., a ring, a star, a mesh,etc.). In the topology, each node has an upstream neighboring node and adownstream neighboring node. Each node is connected to that node'sdownstream neighboring node via a separate unidirectional communicationlink. Each node uses the communication links for sending packets (ormessages, bit sequences, etc.) from that node to that node's downstreamneighboring node. Each node is also connected to that node's upstreamneighboring node via a separate unidirectional control link. Each nodeuses the control link for passing a control token to that node'supstream neighboring node. The control token, which is passed from nodeto node in the topology using the respective unidirectional controllinks, is used by the nodes in operations for ensuring that the nodes inthe topology make forward progress in sending their own packets, asdescribed in more detail below.

In the described embodiments, packets sent by a source node areforwarded from node to node through the topology via respectivecommunication links until reaching an intended destination node. Forexample, a destination node may be two nodes away from a source node inthe topology. In this case, the source node sends the packet to thedestination node by sending the packet to its downstream neighboringnode, which is the first of the intervening nodes, via the respectivecommunication link. The first of the intervening nodes then forwards thepacket to the second of the intervening nodes via the respectivecommunication link. The second of the intervening nodes next forwardsthe packet to the destination node via the respective communicationlink. In this way, the packet progresses through the topology to thedestination node in a series of node-to-node “hops.” In the describedembodiments, the nodes give priority to packets from upstreamneighboring nodes. When a given node has received packets from anupstream neighboring node, therefore, the given node forwards thosepackets in each clock cycle—and will not send its own packets in clockcycles in which packets are forwarded. Without the mechanisms describedherein for controlling the sending of packets by nodes in the topology,“starvation” can occur—i.e., a node can be partially or completelyblocked by upstream nodes from sending its own packets. This can lead toan unfair/undesirable balance of the consumption of bandwidth in thecommunication network—i.e., to one or more nodes using more than adesired (e.g., equal) share of communication network bandwidth.

In some embodiments, one of the operations for ensuring forward progressinvolves using a count of packets sent in each node and respectivethresholds for the count of packets sent to determine if packets arepermitted to be sent by that node and, if so, whether a throttled rateor a full rate is to be used for sending the packets. For thisoperation, a given node increments the count of packets sent uponsending each of the given node's own packets to the downstreamneighboring node via the corresponding communication link, therebykeeping a record of a number of packets sent. As long as the count ofpackets sent is lower than a minimum threshold value (which can be 8,16, or another number) and the given node does not have packets from theupstream neighboring node to be forwarded, the given node can freelysend its own packets during each cycle of a controlling clock (or “clockcycle”). The given node can “freely” send its own packets in that thegiven node is not throttled in sending packets and therefore can sendpackets at the full rate for sending packets. For example, the full ratefor sending packets can be one packet per clock cycle (i.e., a full rateof 1:1). On the other hand, when the count of packets sent is equal toor greater than the minimum threshold value but less than a maximumthreshold value (which can be 32, 64, or another number) and the givennode does not have packets from the upstream neighboring node to beforwarded, the given node sends the given node's own packets at athrottled rate. For the throttled rate, the given node is limited tosending the given node's own packets in less than all clock cycles,i.e., in a specified fraction or proportion of the clock cycles. Forexample, the given node may send packets in one out of every three clockcycles for the throttled rate (i.e., a throttled rate of 1:3). Sendingpackets at the throttled rate avoids the given node sending packets inevery clock cycle, which frees some of the clock cycles for use bydownstream nodes in sending their own packets. When the count of packetssent is equal to the maximum threshold, the given node halts sending itsown packets. The given node does not again send given node's own packetsuntil the count of packets is reset/zeroed (an operation that involvesthe control token, as described below). Halting sending packets meansthat the given node does not send packets in any clock cycle, therebyfreeing all of the clock cycles for use by downstream nodes in sendingtheir own packets.

In the described embodiments, within the communication network, thecontrol token, which can be as simple as a single binary bit, is passedfrom node to node through the topology via respective control links. Inother words, the control token is forwarded from node to node in thetopology, thereby making repeated trips through the nodes in thetopology in a direction in reverse to that of the direction in whichpackets travel in the communication network. Upon receiving the controltoken, each node uses the token to make forward progress in sending thatnode's own packets. More specifically, upon receiving the control token,each node determines whether that node has packets to send—and thus mayhave been facing starvation—and whether that node's count of packetssent is less than the minimum threshold. When that node has no packetsto send or that node's count of packets sent is equal to or greater thanthe minimum threshold, that node passes the control token to theupstream neighboring node via the corresponding control link in the nextclock cycle. That node also, when passing on the control token,zeroes/resets the count of packets sent (which has the effect ofenabling that node to again send that node's own packets, as describedabove). Otherwise, when that node has packets to send and that node'scount of packets sent is less than the minimum threshold, that nodeholds, i.e., does not pass on, the control token. That node then, whileholding the control token, can freely send packets to the downstreamneighboring node via the corresponding communication link. It ispossible that upstream neighboring nodes are (and have been) sendingsufficient packets to that node, however, that that node is busyforwarding packets and cannot send its own packets. By holding thecontrol token, that node prevents upstream nodes from resetting theirown counts of packets sent. The upstream nodes will therefore, if theycontinue to send their own packets, eventually hit the minimumthreshold, leading to the upstream nodes using the throttled rate, andthe maximum threshold, leading to the upstream nodes halting sendingpackets. This provides opportunities for that node to send that node'sown packets in clock cycles that are not consumed in forwarding packetsfrom the upstream nodes. When subsequently sending packets, that nodeincrements that node's own count of packets sent, which eventuallyreaches the minimum threshold value, causing that node to pass on thecontrol token and reset/zero its own count of packets sent. In this way,that node gets some protection from upstream nodes for sending up to aminimum threshold value of its own packets, thereby ensuring forwardprogress for that node.

In some embodiments, the throttled rate is a fixed rate. In theseembodiments, regardless of the value of the count of packets sentbetween the minimum threshold and the maximum threshold, the samethrottled rate is used by each node for sending its own packets. Forexample, the throttled rate may be a constant 1:3 (and thus a given nodeusing the throttled rate may be permitted to send one of its own packetsevery three clock cycles), 1:4, or another rate. In some embodiments,however, the throttled rate is a variable rate that varies in accordancewith the count of packets sent by a given node and/or other criteria. Insome of these embodiments, higher rates are used for lower counts ofpackets sent and lower rates are used for higher counts of packets sent.In other words, in these embodiments, as the count of packets sentincreases, the variable rate is reduced, which can help to avoid a nodestarving downstream nodes and/or consuming more than a desiredproportion of communication network bandwidth. For example, throttledrate of 1:2 may be used for the lowest counts of packets sent (closer tothe minimum threshold) and a throttled rate of 1:8 may be used for thehighest counts of packets sent (closer to the maximum threshold), withone or more intermediate throttled rates possibly being used forintermediate counts of packets sent (e.g., 1:4, 1:6, etc.). In someembodiments, the variable throttled rate is set by a controller in eachnode or another entity including possibly being set for all nodes in thetopology by a “master” node from among the nodes or other entity. Insome embodiments, the variable throttled rate is set based on one ormore operating conditions, packet sending histories, node orcommunication network workload types or priorities, communication linkbusyness or idleness, and/or other criteria for individual nodes orgroups thereof.

In some embodiments, the above-described minimum threshold for using thethrottled rate and the maximum threshold for halting sending packets arefixed. In these embodiments, in other words, the minimum threshold andmaximum threshold are set at respective counts of packets sent and donot change. In some embodiments, however, the minimum threshold and/orthe maximum threshold are dynamically adjustable and can be set atruntime (i.e., as or after the nodes in the communication network havestarted up and are operating). In some embodiments, the minimumthreshold and/or the maximum threshold are set by a controller in eachnode or another entity. For example, in some embodiments, a master nodefrom among the nodes or another entity sets the minimum threshold and/orthe maximum threshold for all nodes in the topology. In someembodiments, the minimum threshold and/or the maximum threshold are setbased on one or more operating conditions, packet sending histories,node or communication network workload types or priorities,communication link busyness or idleness, and/or other criteria forindividual nodes or groups thereof

In some embodiments, the electronic communication network includes twoor more separate topologies, each topology being generally similar tothe above-described topology in terms of nodes, packet communication,and control token passing and handling within that topology. In theseembodiments, the topologies are “separate” in that data is communicatedbetween nodes in the individual topologies, but not between thetopologies themselves. The topologies, however, share a controltoken—and thus a single control token is passed between the topologies.During operation, the control token makes a complete trip though a firstof the topologies so that every node in the first topology receives andhandles the control token as described above. A controller, or“crossover controller,” in a final node of the trip through the firsttopology forwards the control token to the second topology via acrossover control link. A crossover controller in the second topologyreceives the control token from the crossover controller in the firsttopology and starts the control token on a trip through the secondtopology. The control token then makes a complete trip though the secondtopology so that every node in the second topology receives and handlesthe control token as described above. The crossover controller, which isin a final node of the trip through the second topology, forwards thecontrol token back to the first topology via the crossover control link.In this way, the control token passes back and forth between the firsttopology and the second topology, being used to ensure forward progressand fairness for nodes separately in each topology.

By using the count of packets sent and the control token to control whenand how nodes in a topology are able to send their own packets, thedescribed embodiments help to ensure forward progress of nodes (i.e.,avoid starvation and/or ensure fair access for sending packets) withoutunnecessarily hampering the forward progress of other nodes. By passingthe control token between two topologies, the described embodiments helpto ensure forward progress for nodes in both topologies. Ensuringforward progress improves, from the perspective of individual nodes, thecommunication of packets in the topology, which is an improvement to theoverall operation of the communication network. Improved operation ofthe communication network results in increased satisfaction for theusers of the nodes and the communication network.

Node

FIG. 1 presents a block diagram illustrating node 100 in accordance withsome embodiments. Node 100 includes device hardware 102 and networkinterface 104. Device hardware 102 is a functional block that performsoperations such as computational, control, input/output (IO), sensing,and/or other operations in node 100. For example, in some embodiments,device hardware 102 is or includes one or more central processing unit(CPU) cores, graphics processing unit (GPU) cores, embedded processors,and/or application specific integrated circuits (ASICs) that performgeneral purpose computational, control, and/or other operations. Asanother example, device hardware 102 includes data storage and accesscircuit elements and devices such as high-capacity semiconductormemories, disk drives, etc. As yet another example, in some embodiments,device hardware 102 is or includes a sensor such as a video or audiorecording device. Generally, the operations performed by device hardware102, and thus the particular circuit elements and devices found indevice hardware 102, depend on the nature of node 100 (i.e., theparticular type of device of node 100). In the described embodiments,device hardware 102 includes sufficient circuit elements and devices toperform the operations herein described.

Network interface 104 is a functional block that performs operationsassociated with communicating on an electronic communication networksuch as a network that uses the Ethernet standard (e.g., Institute ofElectrical and Electronic Engineers (IEEE) 802.3 and subsequent/relatedstandards) and/or another network standard or protocol. In someembodiments, network interface 104 includes circuit elements and devicesimplementing one or more lower layers, e.g., a physical layer and/or alink layer, of the Open Systems Interconnection (OSI) model, althoughother/higher layers of the OSI model may be implemented in networkinterface 104 some embodiments. Network interface 104 includescontroller 106, which is a functional block such as a microcontroller,an ASIC, etc. that includes circuit elements and devices performoperations such as handling setup and configuration of network interface104 and/or device hardware 102, establishing and maintainingcommunication with other nodes on the communication network,transmitting and receiving communication units associated with thenetwork standard in use (herein called “packets,” although other formsof communication unit can be used), transmitting, receiving, andprocessing a control token, etc. For example, in some embodiments,controller 106 assembles and prepares packets from higher layers of theOSI model (e.g., provided by device hardware 102) for transmission on acommunication link, receives packets via the communication link andprepares the packets for processing in the higher layers of the OSImodel, etc.

Controller 106 is connected to a unidirectional communication link and aunidirectional control link in each of the upstream direction (i.e.,communications (COMM) link 108 and control (CTRL) link 110) and thedownstream direction (i.e., communications (COMM) link 112 and control(CTRL) link 114). In FIG. 1, communication links 108 and 112 are shownusing a heavier line than control links 110 and 114. One reason forusing the heavier line is to enable easy distinction betweencommunication links and control links (particularly with respect to FIG.2). Another reason for using the heavier lines is to illustrate that, insome embodiments, communication links 108 and 112 are multi-wire ormulti-channel communication links such as parallel communication links.Multi-bit communication links are not a requirement, however, andcommunication links 108 and 112 can be single-wire or single-channelcommunication links.

In some embodiments, communication links 108 and 112 and control links110 and 114 are “unidirectional,” which means that traffic (i.e.,packets on the communication links and control tokens on the controllinks) is only permitted to travel in one direction on the links. As canbe seen from the arrows in FIG. 1, in the upstream direction (i.e.,toward an upstream neighboring node of node 100 in a communicationnetwork) communication link 108 is inbound and control link 110 isoutbound. Packets are therefore received from the upstream neighboringnode via communication link 108 and a control token is sent to theneighboring node via control link 110—but packets and the control tokendo not travel in the opposite directions.

Controller 106 maintains a count of packets sent 116 (e.g., in adedicated controller register, etc.). Count of packets sent 116 is aresettable running count of node 100's own packets that have been sentto the downstream neighboring node via communication link 112. Each timethat network interface 104 sends a packet to the downstream neighboringnode, controller 106 increments count of packets sent 116. Controller106 then zeroes/resets count of packets sent 116 when passing thecontrol token to the upstream neighboring node via control link 110.Controller 106 uses count of packets sent 116 to determine if, and atwhat rate, node 100's own packets are permitted to be sent to thedownstream neighboring node via communication link 112. The use of countof packets sent 116 is described in more detail below.

Although shown with a particular number and arrangement of functionalblocks, in some embodiments, node 100 includes different or differentlyarranged functional blocks. For example, in some embodiments, node 100includes a power subsystem, a human interface subsystem (e.g., display,keyboard, mouse, etc.), a memory subsystem, etc. Generally, in thedescribed embodiments, node 100 includes sufficient functional blocks toperform the operations described herein.

Node 100 can be, or can be included in, any electronic device thatperforms memory access and/or other operations. For example, node 100can be, or can be included in, electronic devices such as desktopcomputers, laptop computers, wearable electronic devices, tabletcomputers, smart phones, servers, artificial intelligence apparatuses,virtual or augmented reality equipment, network appliances (e.g.,switches, routers, network-attached storage, etc.), toys, audio-visualequipment, home appliances, controllers, sensors, etc., and/orcombinations thereof.

Topology

In the described embodiments, multiple nodes in an electroniccommunications network are arranged in a topology. In other words, thenodes are connected to one another in a specified pattern. FIG. 2presents a block diagram illustrating nodes 202-210 in a topology 200 inaccordance with some embodiments. Note that the topology shown in FIG. 2is presented as a general example of a topology in some embodiments. Thetopology in other embodiments includes a different number or arrangementof nodes. In some embodiments, nodes 202-210 may, but are not requiredto, have similar internal elements to node 100.

For topology 200, nodes 202-210 are arranged in a ring. Within topology200, each node has an upstream neighboring node and a downstreamneighboring node. For example, node 204 is the downstream neighboringnode of node 202 and node 210 is the upstream neighboring node of node202. As another example, node 208 is the downstream neighboring node ofnode 206 and node 204 is the upstream neighboring node of node 206.

Each of nodes 202-210 is connected to that node's upstream anddownstream neighboring nodes via a separate unidirectional communicationlink. Packets are, as shown in FIG. 2, sent and forwarded downstream intopology 200 via respective communication links. In other words, packetsonly travel in a single direction in topology 200—the downstreamdirection. A packet sent from a source node and destined for adestination node in topology 200 will be received at each interveningnode via the respective inbound communication link from that interveningnode's upstream neighboring node and forwarded to a next downstreamintervening node or the destination node via the respectivecommunication link. For example, assuming that node 202 is the sourcenode and node 208 is the destination node, node 202 initially sends apacket to node 204 via the unidirectional communication link connectedbetween nodes 202 and 204. Node 204 receives the packet and thenforwards the packet to node 206 via the unidirectional communicationlink connected between nodes 204 and 206. Node 206 receives the packetand then forwards the packet to destination node 208 via theunidirectional communication link connected between nodes 204 and 206.Node 208 receives and processes the packet. In this way, packets travelaround topology in a series of “hops” from node to node. In someembodiments, the respective downstream neighboring node is the only nodein the topology to which each node can send packets.

Each of nodes 202-210 is connected to that node's upstream anddownstream neighboring nodes via a separate unidirectional control link.A control token is, as shown in FIG. 2, passed upstream in topology 200via respective control links. In other words, the control token onlytravels in a single direction in topology 200—the upstream direction(which is reverse of the direction in which packets travel in topology200). The control token, which can be as simple as a single binary bitor more complicated, such as being included in a packet, sequence ofbits, or message having a specified format, is passed from node to nodethrough the topology via respective control links, making repeated tripsor loops through topology 200. That is, the control token is handed fromnode to node at a maximum rate of one node per N cycles of a controllingclock (or “clock cycles”) (where Nis a number such as 1, 3, etc.),thereby making trips through the nodes in topology 200 in a specified(upstream) order. When the control token is received at a given node viathe respective unidirectional control link, that node determines whetherthe control token should be held to allow that node to make forwardprogress in sending its own packets, as described in more detail below.In some embodiments, the respective upstream neighboring node is theonly node in the topology to which each node can pass the control token.

Note that “upstream” and “downstream” as used herein are simplyreferences for traffic direction that are used for identifyingdirections from which packets and the control token are inbound and towhich the packets and the control token are outbound at a given node. Insome embodiments, the directions are reversed, so that packets travelupstream and the control token travels downstream. Generally, in thedescribed embodiments, the control token is passed around topology 200in a direction in reverse to that in which packets travel. In addition,although a ring topology is presented in FIG. 2, in some embodiments, adifferent topology is used. For example, a mesh topology, a startopology, a fully-connected topology, etc. can be used—with the topologyhaving specified communication links and control links used forcommunicating packets and passing the control token. Also, in someembodiments, the “unidirectional” communication and/or control links maybe capable of bidirectional traffic, but may be logically or physically(using circuit elements, etc.) restricted to being usedunidirectionally.

Process for Sending Packets by a Node in a Topology

In the described embodiments, nodes in a topology (e.g., nodes 202-210in topology 200) send and forward packets to other nodes in the topologyvia unidirectional communication links. FIG. 3 presents a flowchartillustrating a process for sending packets from a node in a topology inaccordance with some embodiments. Note that the operations shown in FIG.3 are presented as a general example of operations performed by someembodiments. The operations performed by other embodiments includedifferent operations, operations that are performed in a differentorder, and/or operations that are performed by different entities orfunctional blocks.

In the flowchart in FIG. 3, following step 310, the flowchart proceedsto FIG. 4 (which illustrates operations performed by a node in topology200 upon being passed a control token). Proceeding to FIG. 4 as shownillustrates operations after a count of packets sent has beenincremented to a maximum threshold value—and the node halts sendingpackets until the count of packets sent is zeroed/reset. Although thisis the only step in FIG. 3 from which operation is shown as proceedingto FIG. 4, in some embodiments, it is possible to proceed to FIG. 4 fromother steps in FIG. 3. For example, it may be that no packets to be sentare encountered, as shown in steps 304-306, before the control token ispassed to the node, and thus these embodiments may proceed to FIG. 4from step 306.

For the example in FIG. 3, it is assumed that the count of packets sentis initially less than the minimum threshold and thus packets can befreely sent by a node. It is also assumed that the node has at least onepacket to be sent. For example, device hardware 102 may have generatedand provided at least one packet to be communicated to a specifieddestination node. It is further assumed that no packets from aneighboring upstream node are received by the node during the operationsin FIG. 3—and thus no packets from upstream nodes are to be forwarded bythe node. For the example in FIG. 3, therefore, the node is notprevented from sending its own packets due to the need for forwardingpackets received from the upstream neighboring node.

As can be seen in FIG. 3, the process starts when a node in a topologysends a packet to a downstream neighboring node via a unidirectionalcommunication link (step 300). For this operation, a network interface(e.g., network interface 104) in the node receives a packet to be sent,prepares the packet for sending (e.g., by performing operations of oneor more lower layers of an OSI model), and sends the packet to thedownstream neighboring node via the unidirectional communication link.For step 300, the node sends the packet at a full rate. Sending packetsat the full rate involves sending a packet in each cycle of acontrolling clock. The node therefore sends the packet in the next clockcycle (recalling that, for the example in FIG. 3, no packets fromupstream nodes prevent the sending of the node's packets).

A controller in the node (e.g., controller 106), upon determining thatthe node sent the packet to the downstream neighboring node, incrementsthe count of packets sent (e.g., count of packets sent 116) (step 302).Recall that the count of packets sent is a resettable running count ofthat node's own packets that have been sent to the downstreamneighboring node. By incrementing the count of packets sent, therefore,the controller records that another packet has been sent by the node.

The controller then determines whether there are additional packets tobe sent from the node to the downstream neighboring node (step 304). Forexample, the controller may check a level of a packet queue or buffer,communicate with device hardware (e.g., device hardware 102) regardingpackets to be sent, and/or otherwise determine whether the node haspackets to be sent. If not, i.e., if no packets are ready to be sent,the controller waits for packets to be sent (step 306).

When there are packets ready to be sent (step 304), the controllerdetermines whether the count of packets sent is equal to a maximumthreshold value (step 308). For this operation, the controller comparesthe count of packets sent to the maximum threshold value (e.g., a valuesuch as 16, 50, etc.) to determine if they are equal. When the count ofpackets sent equals the maximum threshold value, the controller causesthe node (i.e., the network interface) to halt sending packets (step310). In other words, despite the node having packets ready to be sent,because the count of packets that were already sent is equal to themaximum threshold, the controller stops the node from sending packets.In this way, the controller can help to prevent the node from sending asufficient number of packets to starve a downstream node and/or consumemore than a specified share of packet-sending bandwidth in thecommunication network—and thus free the downstream node to send packetsin clock cycles in which the node has not sent packets. Once thecontroller has halted sending packets, the controller waits for thecontrol token to be passed to the node so that the count of packets sentcan be reset. Operations involving the control token are shown in FIG.4, to which the node proceeds following step 310.

When there are packets to be sent (step 304) and the count of packetssent is less than the maximum threshold value (step 308), the controllerdetermines whether the count of packets sent is greater than or equal toa minimum threshold value (step 312). For this operation, the controllercompares the count of packets sent to the minimum threshold value (e.g.,a value such as 8, 20, etc.) to determine if the count of packets sentis equal to or greater than the minimum threshold. When the count ofpackets sent is equal to or greater than the minimum threshold value,the controller causes the node (i.e., the network interface) to send thepacket at a throttled rate (step 314). Generally, for sending packets atthe throttled rate, the node sends a packet in fewer than each cycle ofa controlling clock when a packet is available to be sent. For example,the node may send a packet every other clock cycle (a 1:2 throttledrate), every fourth clock cycle (a 1:4 throttled rate), or at adifferent rate. In other words, the controller causes the node to skipsending packets in specified clock cycles and send packets in otherclock cycles. By not sending packets in the specified clock cycles, thenode frees the specified clock cycles for use by downstream nodes insending their own packets.

In some embodiments, sending packets at the throttled rate involvessending packets at variable rates (in terms of clock cycles per packet)based on the count of packets sent and/or other criteria. In theseembodiments, the controller performs an additional operation todetermine the throttled rate for sending the packet using the count ofpackets sent. In some embodiments, higher rates (and thus less clockcycles per packet) are used for lower counts of packets sent and lowerrates are used for higher counts of packets sent. For example, for countvalues closer to the minimum threshold value, a 1:2 throttled rate maybe used, whereas a rate of 1:8 is used for count values closer to themaximum threshold value. In these embodiments, as a node sends ever morepackets via the corresponding communication link, the node initiallysends at a higher throttled rate, but eventually begins using a lowerthrottled rate. By not sending packets in gradually higher numbers ofclock cycles, the node frees larger numbers of clock cycles for use bydownstream nodes in sending their own packets. In some embodiments, thenode uses three or more throttled rates in steps between a highestthrottled rate and a lowest throttled rate. For example, in someembodiments, assuming that the minimum threshold value is 10 and themaximum threshold value is 50, between 10-20 count of packets sent, a1:2 throttled rate is used, between 21-30 count of packets sent, a 1:5throttled rate is used, between 31-40 count of packets sent, a 1:8throttled rate is used, and between 41-50 count of packets sent, a 1:12throttled rate is used.

In some embodiments, although not shown in FIG. 3, the controller and/oranother entity (e.g., device hardware in the node, an operating systemor software application executed by the device hardware, a remote nodeor device, etc.) dynamically sets and/or adjusts one or more of thethrottled rate, the maximum threshold, or the minimum threshold. In someof these embodiments, the controller uses specified values (e.g., storedin a memory in the node, provided to the node by the device hardwareand/or the entity, etc.) to initially set the throttled rate, themaximum threshold, and/or the minimum threshold. During operation inthese embodiments, the controller adjusts some or all of the throttledrate, the maximum threshold, and/or the minimum threshold based on oneor more criteria or upon receiving a request to do so from the devicehardware and/or the entity. For example, the controller, the devicehardware, and/or the other entity may keep a history of clock cycles ofdelay in sending packets (i.e., how long the node and/or other nodeshave been halted and/or using the throttled rate(s)), the busyness oridleness of the communication links in the topology, the depth of packetqueues or buffers in the nodes, the type of workloads being executed inthe node and/or other nodes, the characteristics of packets being sent(e.g., type, size, sequence numbers, etc.), and/or other informationabout the node, other nodes, and/or the communication network, and mayuse one or more rules, algorithms, mathematical expressions, etc. tocompute or determine one or more of the throttled rate, the maximumthreshold, and/or the minimum threshold to be used by the controller forthe adjustment.

Returning to FIG. 3, when the count of packets sent is less than theminimum threshold value (Step 312), the process returns to step 300,where the node sends the packet at the full rate.

Process for Handling a Control Token by a Node in a Topology

In the described embodiments, a control token is passed from node tonode in a topology (e.g., nodes 202-210 in topology 200) via respectiveunidirectional control links and used by nodes for controlling thesending of packets. FIG. 4 presents a flowchart illustrating a processfor handling a control token in a node in a topology in accordance withsome embodiments. Note that the operations shown in FIG. 4 are presentedas a general example of operations performed by some embodiments. Theoperations performed by other embodiments include different operations,operations that are performed in a different order, and/or operationsthat are performed by different entities or functional blocks.

For the example in FIG. 4, it is assumed that no packets from aneighboring upstream node are received by the node during the operationsin FIG. 4—and thus no packets from upstream nodes are to be forwarded bythe node. For the example in FIG. 4, therefore, the node is notprevented from sending its own packets due to the need for forwardingpackets received from the upstream neighboring node. In addition, forthe example in FIG. 4, a control token is assumed to be a single binarybit. “Passing” the control token from a given node therefore meanssignaling a value of a single binary bit to the given node's downstreamneighboring node via the respective control link. For example, drivercircuit elements in each node may maintain the control link in a logicallow state (e.g., a logical zero, VDD in an integrated circuit, etc.)when the control token is not being passed, and may temporarily/brieflydrive the control link to a logical high state for passing the controltoken (e.g., a logical one, VSS in an integrated circuit, etc.).

As can be seen in FIG. 4, the process starts when a node in a topologyreceives a control token from a downstream neighboring node via acorresponding unidirectional control link (step 400). For thisoperation, a network interface (e.g., network interface 104) in the nodemonitors for the control token, such as by sensing the level of a signalon the control link, and detects/receives the control token accordingly.

A controller in the node (e.g., controller 106) then determines whetherthe node has packets to be sent (step 402). For example, the controllermay check a level of a packet queue or buffer, communicate with devicehardware (e.g., device hardware 102) regarding packets to be sent,and/or otherwise determine whether the node has packets to be sent. Whenthe node does not have packets to be sent, the controller zeroes/resetsa count of packets sent (step 404). Recall that the count of packetssent is a resettable running total of packets sent to a downstreamneighboring node via a corresponding communication link. Zeroing thecount of packets sent therefore sets the count of packets sent toindicate that no packets have been sent from the node—since the nodelast held the control token (or in an initial state). By zeroing thecount of packets sent, therefore, the controller resets the node so thatthe node is able to again send packets at a full rate. Because the nodemay have been halted from sending packets (as described above for steps308-310) or sending packets at the throttled rate (as described abovefor steps 312-314), this operation restores the node to full-ratesending of packets. The node then passes the control token to anupstream neighboring node via the corresponding unidirectional controllink (step 406). Once the controller has passed the control token to theupstream neighboring node, the node commences sending packets.Operations for sending packets are shown in FIG. 3, to which the nodeproceeds following step 406.

When the node has packets to be sent (step 402), the controllerdetermines whether the count of packets sent is equal to or greater thana minimum threshold (step 408). If the count of packets sent is equal toor greater than the minimum threshold value, i.e., if the node has sentat least the minimum threshold value worth of its own packets to adownstream neighboring node, the node passes the control token to theupstream neighboring node. For this operation, the node proceeds to step404 and performs the subsequent operations as described above.

When the node has packets to be sent (step 402) and the count of packetssent is less than the minimum threshold value (step 408), the controllercauses the node to hold the control token (step 410). As used herein,“holding” the control token indicates that the node does not immediatelypass the control token (i.e., assert the associated signal on thecontrol link to the upstream neighboring node in the next clock cycle),but instead waits to pass the control token in order to send at leastone of the node's own packets. The node then sends a packet to thedownstream neighboring node (step 412). For this operation, the nodesends the packet to the downstream neighboring node via thecorresponding communication link at the full rate. The node then returnsto step 402 to determine if the node has more of the node's own packetswaiting to be sent.

Process for Forwarding Packets by a Node in a Topology

In the described embodiments, nodes in a topology forward packetsreceived from upstream neighboring nodes. FIG. 5 presents a flowchartillustrating forwarding packets from a node in accordance with someembodiments. Note that the operations shown in FIG. 5 are presented as ageneral example of operations performed by some embodiments. Theoperations performed by other embodiments include different operations,operations that are performed in a different order, and/or operationsthat are performed by different entities or functional blocks.

The process in FIG. 5 starts when a node in a topology receives, from anupstream neighboring node via the corresponding unidirectionalcommunication link, a packet that is destined for an other node in thetopology (step 500). For example, node 204 in topology 200 can receive,from node 202, via the corresponding communication link, a packet thatis destined for node 206. The node receives the packet via receivingcircuit elements such as a receiving buffer, etc. The node then forwardsthe packet to a downstream neighboring node via the correspondingunidirectional communication link in a cycle of the controlling clock(step 502). In other words, in a next available clock cycle, the nodeforwards the received packet to the downstream neighboring node.

The node also blocks the sending of the node's own packets during thecycle of the controlling clock (step 504). For this operation, the nodegives preference to forwarding the received packet over sending its ownpackets, as is a policy in the topology/communication network. Asdescribed above, without the mechanisms described herein, the operationsin FIG. 5 can result in the starvation of the node and/or anunfair/undesirable balance of the consumption of bandwidth in thecommunication network when the upstream node sends sufficient packets tothe node to prevent the node from sending some or all of its ownpackets.

Packet and Control Token Communication in a Topology

In the described embodiments, packets are sent from nodes in a topologyto other nodes in the topology to communicate data, control values,and/or other information. In addition, a control token is passed fromnode to node in the topology and used by each node to control thesending of packets by the nodes. FIG. 6 presents a swim lane diagramillustrating communications of nodes in a topology in accordance withsome embodiments. Note that the communications shown in FIG. 6 are ageneral example of communications in accordance with some embodiments.In other embodiments, however, different communications and/or differentcriteria (e.g., full rate, throttled rate, maximum threshold value,minimum threshold value, etc.) are used.

For the example in FIG. 6, nodes 0-2, i.e., N0, N1, and N2, are locatedin a topology similarly to nodes 202-206 in topology 200, with N0 beinglocated similarly to node 206, N1 being located similarly to node 204,and N2 being located similarly to node 202. In other words, N0 and N1are connected via unidirectional communication and control links with N0downstream from N1 akin to nodes 204 and 206. N1 and N2 are alsoconnected via unidirectional communication and control links with N2downstream from N1 akin to nodes 204 and 206. Differently than topology200, however, N2 and N0 are connected to one another via correspondingunidirectional communication and control links so that N2 is downstreamfrom N0 (in terms of packet flow). In other words, N2-N1-N0 are arrangedin a ring topology.

For the example in FIG. 6, it is assumed that N1 and N2 both have alarger number of packets to be sent to N0 and N0 has no packets of itsown to send. In addition, it is assumed that packets are sentdownstream, i.e., from N2 to N1 to N0, and the control token is passedupstream, i.e., from N0 to N1 to N2. In order to communicate packets toN0, therefore, N2 sends packets to N1, which then forwards the packetsto N0. The sending and forwarding of the packets is shown using theblack solid arrows and the passing of the control token is shown usingheavier dashed lines. The packets and the control token each proceedfrom node to node in corresponding clock cycles, i.e., based on levelsand/or transitions of a controlling clock signal for the nodes thatoscillates between a logical high level and a logical low level in eachcycle of a repeating sequence of cycles. A number of clock cycles arenumbered and shown using lighter dashed lines on the left side of FIG.6—as is the flow of time. For example, in clock cycle 1, a packet issent from N1 to N0, as shown by the angled black solid arrow—i.e., thepacket leaves N1 sometime during clock cycle 1 and is received in N0 byapproximately the end of clock cycle 1.

For the example in FIG. 6, the minimum threshold used for determininghow packets are allowed to be sent is set to a value of 4. The maximumthreshold used for determining how packets are allowed to be sent is setto a value of 8. These values for the thresholds are merely examples andsome embodiments use different values. In addition, a full rate forsending packets is a packet per clock cycle and a throttled rate forsending packets is a packet every other clock cycle (i.e., a throttledrate of 1:2). These values for the full rate and the throttled rate aremerely examples and some embodiments use different values.

The communications in FIG. 6 start when each of N1 and N2 sends a packetdestined for N0 via the respective unidirectional communication links ina first clock cycle. As described above, N2 sends the packet to N1 sothat N1 can forward the packet to N0 in a second clock cycle. Uponsending the packets, a controller in each of N1 and N2 increments arespective count of packets sent, which is shown to the right of theswim lane line for each of nodes N1 and N2 (a few values of N2's countof packets sent are labeled in FIG. 2). In addition, in the first clockcycle, N0 passes the control token to N1.

Upon receiving the control token, a controller in N1 determines boththat: N1 has its own packets to send and N1's count of packets sent isbelow the minimum threshold. N1 therefore holds the control token, i.e.,does not immediately pass the control token to N2. The holding of thecontrol token is shown via a cross-hatched bar in the swim lane for N1.By holding the control token, N1 prevents N2 from resetting N2's countof packets sent.

In clock cycles 2-5, N2 sends packets to N1 to be forwarded to N0.Because the count of packets sent for N2 is lower than the minimumthreshold in these cycles, N2 sends packets at the full rate, i.e., onepacket per clock cycle. N2 also increments N2's count of packets sent ineach clock cycle in which a packet is sent. Because N1 receives andforwards packets from N2 in cycles 2-5, N1 is unable to send N1's ownpackets, but continues to hold the control token. In clock cycles 5-12,because N2's count of packets sent is greater than the minimumthreshold, N2 sends packets at the throttled rate. N2 therefore sendspackets in clock cycles 7, 9, and 11—incrementing N2's count of packetssent accordingly. In clock cycles where a packet is not received fromN2, i.e., clock cycles 6, 8, and 10, N1 is able to send N1's ownpackets. N1 increments N1's count of packets sent in each clock cycle inwhich N1 sends a packet.

In clock cycle 12, after sending a packet, N2 increments N2's count ofpackets sent to a value equal to the maximum threshold. Because N2'scount of packets sent equals the maximum threshold, N2 halts sendingpackets (shown via the label HALTED in FIG. 6). Also in clock cycle 12,N1 sends sufficient packets to increment N1's count of packets sent tothe minimum threshold value, which means that N1 zeroes/resets N1'scount of packets sent and passes the control token to N2.

Upon receiving the control token, N2 determines that the count ofpackets sent is greater than the minimum threshold value, and thus thecontrol token is not to be held. N2 therefore zeroes/resets N2's countof packets sent and passes the control token to N0 (which is shown onthe left and right sides of FIG. 6 for clarity). When N2 zeroes/resetsN2's count of packets sent, it enables N2 to again send packets at thefull rate, which N2 does from clock cycles 15-19, incrementing N2'scount of packets sent. In clock cycle 19, N2 again has a count ofpackets sent that is equal to the minimum threshold value, so N2commences sending packets at the throttled rate. As N2 sends packets atthe throttled rate, N1 is able to send packets in alternating clockcycles.

Control Token Crossover in Multi-Topology Communication Network

In some embodiments, the above-described mechanisms for using a controltoken to ensure forward progress for nodes in topology in acommunication network are extended to nodes in multiple topologies. FIG.7 presents a block diagram illustrating nodes 700-706 in topology 708and nodes 710-716 in topology 718 in accordance with some embodiments.Note that the topologies in FIG. 7 and the operations performed by nodestherein are examples of some embodiments. Other embodiments usedifferent topologies (and perhaps two different topologies) and/orperform different operations.

The individual topologies 708 and 718 in FIG. 7 each behave similarly totopology 200 in terms of packets being sent and forwarded downstream andcontrol tokens being passed upstream. In addition, a controller in eachnode keeps and uses a count of packets sent to determine if and at whatrate that node's own packets are to be sent to a downstream neighboringnode. The controller in each node also uses the control token asdescribed above—i.e., holds the control token, zeroes the count ofpackets sent when passing on the control token, etc. In other words,each of the topologies passes the control token from node to node ontrips through the topology so that the control token makes completetrips through every node of the topology. This is shown by topology trip724 in topology 708 and topology trip 726 in topology 718.

Differently than topology 200, the control token is, at the end of eachcomplete trip through each topology, sent to the other topology in orderto make a complete trip through that topology. For example, at the endof a topology trip 724 in topology 708 (i.e., through nodes702-700-706-704 in sequence), the control token is passed by a crossovercontroller in node 702 to a crossover controller in node 716 viatopology crossover 720. The control token then makes a trip throughtopology 718. Then, at the end of a topology trip 726 in topology 718(i.e., through nodes 716-714-712-710 in sequence), the control token ispassed by a crossover controller in node 716 to a crossover controllerin node 702 via topology crossover 722. In this way, the control tokenpasses back and forth between topology 708 and topology 718, separatelycontrolling operations in each topology.

Note that the topology that does not have the control token functions asdescribed above with regard to sending packets based on counts ofpackets sent. In other words, nodes in that topology will continue tosend packets at the full rate and then throttled rate until therespective count of packets sent reaches the maximum threshold value. Insome cases, the count of packets sent in nodes will reach the maximumthreshold value while the control token is in the other topology—andsuch nodes will halt sending packets until the control token returns tothat topology and that node. In addition, in some embodiments, thethreshold values are configured in view of the travel of the controltoken between both topologies 708 and 718 and the fairness of bandwidthconsumption (i.e., in terms of the percentage of bandwidth consumptionby each node) within each topology. For example, in some embodiments,the minimum threshold is set equal to the maximum threshold, so thatnodes in topologies 708 and 718 are either sending packets at the fullrate or are halted when the respective count of packets sent reaches themaximum threshold—and thus the nodes do not use the throttled rate.

In some embodiments, at least one electronic device (e.g., node 100,controller 106, etc.) uses code and/or data stored on a non-transitorycomputer-readable storage medium to perform some or all of theoperations herein described. More specifically, the at least oneelectronic device reads code and/or data from the computer-readablestorage medium and executes the code and/or uses the data whenperforming the described operations. A computer-readable storage mediumcan be any device, medium, or combination thereof that stores codeand/or data for use by an electronic device. For example, thecomputer-readable storage medium can include, but is not limited to,volatile and/or non-volatile memory, including flash memory, randomaccess memory (e.g., eDRAM, RAM, SRAM, DRAM, DDR4 SDRAM, etc.),non-volatile RAM (e.g., phase change memory, ferroelectric random accessmemory, spin-transfer torque random access memory, magnetoresistiverandom access memory, etc.), read-only memory (ROM), and/or magnetic oroptical storage mediums (e.g., disk drives, magnetic tape, CDs, DVDs,etc.).

In some embodiments, one or more hardware modules perform the operationsherein described. For example, the hardware modules can include, but arenot limited to, one or more processors/cores/central processing units(CPUs), application-specific integrated circuit (ASIC) chips, neuralnetwork processors or accelerators, field-programmable gate arrays(FPGAs), compression and/or encoding subsystems, compute units, embeddedprocessors, graphics processors (GPUs)/graphics cores, acceleratedprocessing units (APUs), functional blocks, controllers, accelerators,and/or other programmable-logic devices. When circuitry (e.g.,integrated circuit elements, discrete circuit elements, etc.) in suchhardware modules is activated, the circuitry performs some or all of theoperations. In some embodiments, the hardware modules include generalpurpose circuitry such as execution pipelines, compute or processingunits, etc. that, upon executing instructions (program code, firmware,etc.), perform the operations. In some embodiments, the hardware modulesinclude purpose-specific or dedicated circuitry that performs theoperations, possibly including circuitry that performs some or all ofthe operations without executing instructions.

In some embodiments, a data structure representative of some or all ofthe functional blocks and circuit elements described herein (e.g., node100, controller 106, or some portion thereof) is stored on anon-transitory computer-readable storage medium that includes a databaseor other data structure which can be read by an electronic device andused, directly or indirectly, to fabricate hardware including thefunctional blocks and circuit elements. For example, the data structuremay be a behavioral-level description or register-transfer level (RTL)description of the hardware functionality in a high level designlanguage (HDL) such as Verilog or VHDL. The description may be read by asynthesis tool which may synthesize the description to produce a netlistincluding a list of transistors/circuit elements from a synthesislibrary that represent the functionality of the hardware including theabove-described functional blocks and circuit elements. The netlist maythen be placed and routed to produce a data set describing geometricshapes to be applied to masks. The masks may then be used in varioussemiconductor fabrication steps to produce a semiconductor circuit orcircuits (e.g., integrated circuits) corresponding to theabove-described functional blocks and circuit elements. Alternatively,the database on the computer accessible storage medium may be thenetlist (with or without the synthesis library) or the data set, asdesired, or Graphic Data System (GDS) II data.

In this description, variables or unspecified values (i.e., generaldescriptions of values without particular instances of the values) arerepresented by letters such as N, M, and X As used herein, despitepossibly using similar letters in different locations in thisdescription, the variables and unspecified values in each case are notnecessarily the same, i.e., there may be different variable amounts andvalues intended for some or all of the general variables and unspecifiedvalues. In other words, N and any other letters used to representvariables and unspecified values in this description are not necessarilyrelated to one another.

The expression “et cetera” or “etc.” as used herein is intended topresent an and/or case, i.e., the equivalent of “at least one of” theelements in a list with which the etc. is associated. For example, inthe statement “the electronic device performs a first operation, asecond operation, etc.,” the electronic device performs at least one ofthe first operation, the second operation, and other operations. Inaddition, the elements in a list associated with an etc. are merelyexamples from among a set of examples—and at least some of the examplesmay not appear in some embodiments.

The foregoing descriptions of embodiments have been presented only forpurposes of illustration and description. They are not intended to beexhaustive or to limit the embodiments to the forms disclosed.Accordingly, many modifications and variations will be apparent topractitioners skilled in the art. Additionally, the above disclosure isnot intended to limit the embodiments. The scope of the embodiments isdefined by the appended claims.

What is claimed is:
 1. A communication network, comprising: a plurality of nodes configured in a topology, each node having an upstream neighboring node and a downstream neighboring node in the topology; a separate unidirectional communication link coupled between each node and that node's downstream neighboring node, wherein packets sent from nodes in the topology are forwarded from node to node via corresponding unidirectional communication links until reaching destination nodes; a separate unidirectional control link coupled between each node and that node's upstream neighboring node, wherein a control token is passed from node to node in the topology via the unidirectional control links, the control token being used to ensure forward progress in sending packets from the nodes; and a controller in each node, the controller in each node: keeping a count of packets sent by that node; using the count of packets sent to determine whether a given packet is allowed to be sent from that node to the downstream neighboring node and, if so, whether a full rate or a throttled rate is to be used for sending the given packet; and based at least in part on the determining, selectively sending the given packet to the downstream neighboring node.
 2. The communication network of claim 1, wherein the controller in each node determines, based on the count of packets sent, that: packets are not allowed to be sent, and thus sending packets is to be halted, when the count of packets sent is equal to a maximum threshold; packets are to be sent at the throttled rate when the count of packets sent is equal to or greater than a minimum threshold but less than the maximum threshold; and packets are to be sent at the full rate when the count of packets sent is less than the minimum threshold.
 3. The communication network of claim 2, wherein: sending packets at the full rate involves sending a packet in each cycle of a controlling clock when a packet is available to be sent; and sending packets at the throttled rate involves sending a packet in fewer than each cycle of a controlling clock when a packet is available to be sent.
 4. The communication network of claim 3, wherein sending a packet in fewer than each cycle of the controlling clock includes: sending packets at variable rates in terms of clock cycles per packet based at least in part on a value of the count of packets sent between the minimum threshold and the maximum threshold, wherein: higher rates are used when the value of the count of packets sent is closer to the minimum threshold; and lower rates are used when the value of the count of packets sent is closer to the maximum threshold.
 5. The communication network of claim 2, wherein the controller in each node or another entity dynamically sets at least one of the respective minimum threshold and the maximum threshold based on one or more operating criteria.
 6. The communication network of claim 1, wherein, for keeping a count of packets sent by each node, the controller in that node: increments a respective count of packets sent upon sending each packet; and sets the respective count of packets sent to zero upon passing the control token to the upstream neighboring node.
 7. The communication network of claim 1, wherein the controller in each node is configured to: receive packets destined for an other node from that node's upstream neighboring node via the unidirectional communication link between the upstream neighboring node and that node; and forward the packets to that node's downstream neighboring node via the unidirectional communication link between that node and the downstream neighboring node; wherein, while forwarding packets received from the upstream neighboring node, that node is unable to send that node's own packets to the downstream neighboring node.
 8. The communication network of claim 1, wherein each controller, upon being passed the control token from the downstream neighboring node: determines a respective count of packets sent; determines whether that node has packets to be sent; when the respective count of packets sent is equal to or greater than a minimum threshold or the node has no packets to send, that controller: zeroes the respective count of packets sent; and passes the control token to the upstream neighboring node; and when the respective count of packets sent is less than a minimum threshold and the node has packets to send, that controller: holds the control token; and sends packets to the downstream neighboring node via the unidirectional communication link between that node and the downstream neighboring node in cycles of a controlling clock in which that node is not forwarding packets received from the upstream neighboring node.
 9. The communication network of claim 8, wherein: upon sending each packet to a given node that is holding the control token, a particular upstream neighboring node increments a respective count of packets sent; and when the particular upstream neighboring node has sent sufficient packets to increment the respective count of packets sent to be equal to a maximum threshold value, the particular upstream neighboring node halts sending packets, thereby avoiding sending packets to the given node that block the given node from sending the given node's own packets and freeing the given node to make forward progress in sending the given node's own packets.
 10. The communication network of claim 1, wherein the control token is passed from node to node in the topology in a first direction on the unidirectional control links, the first direction being opposite to a second direction in which packets are sent and forwarded between nodes in the topology on the unidirectional communication links.
 11. The communication network of claim 1, wherein the topology is or is included in a ring, a mesh, or a star topology.
 12. A communication network, comprising: a plurality of first topology nodes configured in a first topology, each first topology node having: an upstream neighboring first topology node and a downstream neighboring first topology node in the first topology, a separate unidirectional communication link coupled between that first topology node and that first topology node's downstream neighboring first topology node, and a separate unidirectional control link coupled between that first topology node and that first topology node's upstream neighboring first topology node, wherein packets sent from first topology nodes are forwarded from first topology node to first topology node in the first topology via corresponding unidirectional communication links until reaching destination first topology nodes; a plurality of second topology nodes configured in a second topology, each second topology node having: an upstream neighboring second topology node and a downstream neighboring second topology node in the second topology, a separate unidirectional communication link coupled between that second topology node and that second topology node's downstream neighboring second topology node, and a separate unidirectional control link coupled between that second topology node and that second topology node's upstream neighboring second topology node, wherein packets sent from second topology nodes are forwarded from second topology node to second topology node in the second topology via corresponding unidirectional communication links until reaching destination second topology nodes; and a controller in each first topology node and second topology node, each controller passing a control token from node to node in the respective topology via the unidirectional control links, the control token being used to ensure forward progress in sending packets from the nodes via the respective communication links; wherein a first controller in one of the first topology nodes and a second controller in one of the second topology nodes are crossover controllers that exchange the control token after each trip through the respective topology, so that each crossover controller sends the control token to an other topology of the first topology or the second topology for a trip through the other topology.
 13. The communication network of claim 12, wherein the controller in each first topology node and each second topology node is configured to: keep a count of packets sent by that node; use the count of packets sent to determine whether a given packet is allowed to be sent from that node to that node's downstream neighboring node and, if so, whether a full rate or a throttled rate is to be used for sending the given packet; and based at least in part on the determining, selectively send the given packet to that node's downstream neighboring node.
 14. The communication network of claim 13, wherein the controller in each first topology node and each second topology node determines, based on the count of packets sent, that: packets are not allowed to be sent, and thus sending packets is to be halted, when the count of packets sent is equal to a maximum threshold; packets are to be sent at the throttled rate when the count of packets sent is equal to or greater than a minimum threshold but less than the maximum threshold; and packets are to be sent at the full rate when the count of packets sent is less than the minimum threshold.
 15. The communication network of claim 14, wherein: sending packets at the full rate involves sending a packet in each cycle of a controlling clock when a packet is available to be sent; and sending packets at the throttled rate involves sending a packet in fewer than each cycle of a controlling clock when a packet is available to be sent.
 16. The communication network of claim 15, wherein sending a packet in fewer than each cycle of the controlling clock includes: sending packets at variable rates in terms of clock cycles per packet based at least in part on a value of the count of packets sent between the minimum threshold and the maximum threshold, wherein: higher rates are used when the value of the count of packets sent is closer to the minimum threshold; and lower rates are used when the value of the count of packets sent is closer to the maximum threshold.
 17. The communication network of claim 14, the controller in each node or another entity dynamically sets at least one of the respective minimum threshold and the maximum threshold based on one or more operating criteria.
 18. The communication network of claim 13, wherein, for keeping a count of packets sent by each first topology node and each second topology node, the controller in that node: increments a respective count of packets sent upon sending each packet; and sets the respective count of packets sent to zero upon passing the control token to the upstream neighboring node.
 19. The communication network of claim 13, wherein the controller in each first topology node and each second topology node is configured to: receive packets destined for an other node from that node's upstream neighboring node via the unidirectional communication link between that node's upstream neighboring node and that node; and forward the packets to that node's downstream neighboring node via the unidirectional communication link between that node and that node's downstream neighboring node; wherein, while forwarding packets received from the upstream neighboring node, that node is unable to send that node's own packets to the downstream neighboring node.
 20. The communication network of claim 13, wherein the controller in each first topology node and each second topology node, upon being passed the control token from that node's downstream neighboring node is configured to: determine a respective count of packets sent by that node; determine whether that node has packets to be sent; when the respective count of packets sent is equal to or greater than a minimum threshold or that node has no packets to send, that controller is configured to: zero the respective count of packets sent; and pass the control token to that node's upstream neighboring node; and when the respective count of packets sent is less than a minimum threshold and that node has packets to send, that controller is configured to: hold the control token; and send packets to that node's downstream neighboring node via the unidirectional communication link between that node and that node's downstream neighboring node in cycles of a controlling clock in which that node is not forwarding packets received from that node's upstream neighboring node.
 21. The communication network of claim 20, wherein: upon sending each packet to a given node that is holding the control token, a particular upstream neighboring first topology node or second topology node increments a respective count of packets sent; and when the particular upstream neighboring first topology node or second topology node has sent sufficient packets to increment the respective count of packets sent to be equal to a maximum threshold value, the particular upstream neighboring first topology node or second topology node halts sending packets, thereby avoiding sending packets to the given node that block the given node from sending the given node's own packets and freeing the given node to make forward progress in sending the given node's own packets.
 22. The communication network of claim 12, wherein: the control token is passed from first topology node to first topology node in the first topology in a first direction on the unidirectional control links, the first direction being opposite to a second direction in which packets are sent and forwarded between first topology nodes in the first topology on the unidirectional communication links; and the control token is passed from second topology node to second topology node in the second topology in a second direction on the unidirectional control links, the second direction being opposite to a second direction in which packets are sent and forwarded between second topology nodes in the second topology on the unidirectional communication links.
 23. The communication network of claim 12, wherein the first topology and/or the second topology is or is included in a ring, a mesh, or a star topology. 